Does Reform-Oriented Teaching Make a Difference? The Relationship Between Teaching Practices and Achievement in Mathematics and Science

In a three-year study, RAND researchers examined the relationship between reform-oriented instruction and student performance in mathematics and science. At the end of the study, students who had been exposed to more reform-oriented teaching performed better in both math and science than those who had experienced less, but the differences in scores were small. The relationship between instructional practice and student performance was stronger when performance was measured on items that required problem-solving skills rather than procedural skills, an outcome that is generally consistent with the goals of reform-oriented teaching. These results illustrate the importance of matching performance measures to reform goals in evaluating instructional innovations. In science, students might be expected to generate scientifi c questions, plan research, collect data, analyze relationships, and write reports. More generally, reform-oriented instruction encourages inquiry-based activities and the skills of intellectual conversation: asking questions, discussing alternative approaches to problems, presenting reasons for answers to questions, and making connections between old and new knowledge and between superfi cially disparate topics. Th e study of reform-oriented instruction described here, called Mosaic II, was an extension of Mosaic I, an earlier investigation in which a team of RAND researchers found “a weak but positive relationship” between reform-oriented instructional practices and students’ scores on standardized tests. Intrigued by these results, they undertook a new study focused on the same question, but designed to yield a more detailed picture of this relationship. Th e new study involved three key methodological features: a longitudinal research design, multiple measures of instructional practices, and multiple measures of student performance. Student performance on standardized tests was measured over a three-year period At the outset of the study, the RAND team selected fi ve cohorts of elementary and middle-school students in three school districts. A cohort consisted of all the students in a selected grade in one of the two subject areas (for example, third-grade mathematics students or sixth-grade science students).1 Each cohort was followed for three years, and their performance on standardized tests was measured at the end of each year (current-year performance), as well as aggregated at the end of the three-year period (cumulative performance). Multiple measures were used to assess instructional practice To obtain data regarding teachers’ choices and classroom activities as they occur, the Mosaic II team used the following methods to assess instructional practices: • Teacher Surveys: Th e surveys gathered information about teachers’ educational background, experience, and classroom practices. • Classroom Vignettes: Teachers were presented with written vignettes, consisting of hypothetical classroom situations, and asked to indicate how likely it was that they would choose each of a list of possible actions that could be taken in response. Th e actions refl ected a range of reform-oriented and non-reform-oriented alternatives. Unlike the survey, which required teachers to recall their actions over a wide range of situations over a long period, this method provided teachers with a realistic context in which to situate their responses, potentially off ering insights into their preferred teaching style and increasing the generalizability of their responses. • Teacher Logs: Each day during a twoto fi ve-day period, teachers were asked to fi ll out a log by checking off items on a list describing specifi c activities that occurred during mathematics or science lessons. Th e logs focused on both teacher and student activities each day during the data-collection period. Th e immediacy of this procedure was intended to increase accuracy in reporting of classroom activities. • Classroom Observations: Th e RAND team observed 52 teachers, recording behaviors on a protocol designed to capture important elements of reform-oriented instruction. Because the results of these observations were independent of teacher reports, they were free of any biases due to forgetting or other reporting limitations. Both multiple-choice and open-ended measures were used to assess student performance on standardized tests Two measures—scores on both multiple-choice and openended items on standardized tests—were used as indicators of student learning in mathematics and science. In addition, performance on items that require problem-solving skills was diff erentiated from performance on items that required only the application of procedural skills.